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Abstract 

Using data from a self-report survey of juniors and seniors, we 
document statistically significant growth of a student engagement cul¬ 
ture on our campus after a three year period following implementation 
of an institutional Quality Enhancement Plan. Twenty-three of 45 individ¬ 
ual variables, scores on four of 10 multivariate engagement factors, and 
four of four QEP outcome variables showed significant increases over the 
course of the study. Observed changes reflect institutional priorities and 
intentionality in advancing discrete learning outcomes related to student 
engagement, including greater awareness of key societal issues, enhanced 
capacity to apply their knowledge and skills in addressing societal con¬ 
cerns, increased appreciation of diverse perspectives, and a greater sense 
of social responsibility. Our approach accommodates alternative, interre¬ 
lated approaches to addressing common objectives to better reflect the cu¬ 
mulative, multifactorial nature of student learning, and provides a robust, 
powerful model for institutional-level student learning assessment. 

Introduction 

Assessing the impact of pedagogical initiatives on students’ knowl¬ 
edge, skills, and perspectives is like trying to break open a pinata: we cel¬ 
ebrate each attempt, even though most make little discemable progress, 
yet never really know what combination of factors ultimately contribute to 
the final, desired outcome. As higher education becomes increasingly out¬ 
comes- and accountability-driven, it is essential that we find ways to doc¬ 
ument added value in students’ educational experiences. Recent national 
movements towards student engagement (Kuh, 2001 a), civic and political 
engagement (Colby, Beaumont, Ehrlich & Corngold, 2007; Colby, Eh¬ 
rlich, Beaumont & Stephens, 2003; Mehaffy, 2005), personal and social 
responsibility (Colby & Sullivan, 2009; Hersch & Schneider, 2005), and 
critical thinking (Paul & Elder, 2006), among others, have forced faculty 
and administrators to confront the issue of assessing higher-order learning 
outcomes across the curriculum and co-curriculum. 

Most assessments to date have fallen into one of three catego¬ 
ries. First, process indicators such as the number of learning opportunities 
available and the extent to which students have participated in such activi¬ 
ties have been used to demonstrate increased intentionality by individuals 
and institutions in promoting targeted outcomes; the National Survey of 
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Student Engagement (NSSE) represents the most comprehensive example 
of this type, and relies on studies which show correlations between educa¬ 
tional practice and student achievement (e.g., Astin, 1991; Carini, Kuh & 
Klein, 2006; Chickering & Gamson, 1987; Kuh, 2001a, 2001b). Second, 
pretest-posttest approaches have been used to document the extent of pos¬ 
itive changes in learning and/or attitudes within the context of individual 
courses and experiences (e.g., Eluerta, Jorgensen & Jozwiak, 2006); while 
potentially valuable in demonstrating short-term gains using an hypothe¬ 
sis-testing approach, such assessments cannot reflect cumulative enhance¬ 
ment or even regression of learning gains over time. Finally, portfolios 
and other reflective assessments are becoming more common as means of 
assessing students’ individual growth and capacity (e.g., Donnelly-Smith, 
2008); these approaches offer the richest illustration of the student learn¬ 
ing experience, but may be difficult to translate into quantitative measures 
suitable for institutional assessment. 

A few studies have employed a multivariate statistical approach to 
quantitative assessment. Tan (1992) used factor analysis to derive multi¬ 
dimensional predictors of academic quality based on data from sociology 
departments at 54 U.S. research universities. This study identified fac¬ 
tors related to faculty research productivity, departmental size, and student 
success as contributors to indicators of departmental excellence. Trigwell, 
Prosser & Waterhouse (1999) used factor and cluster analyses to describe 
relationships between teachers’ and students’ approaches to learning in 48 
first-year classes in Australian universities; they found that learning strat¬ 
egies adopted by students are associated with professors’ approaches to 
teaching. Kuh (2001b) used principal components analysis and factor ro¬ 
tation to examine the validity and psychometric properties of the NSSE 
instrument, identifying sets of student engagement constructs consistent 
with those previously linked to effective educational practice. These stud¬ 
ies have exploited statistical correlations among individual variables to 
derive a small number of composite factors and/or clusters that explained 
a majority of the total variation and, in some cases, subsequently test for 
significant differences among groups in multivariate space. 

The statistical power and flexibility i nh erent in multivariate ap¬ 
proaches complement a number of models of cognitive development and 
learning. Perry’s (1970) model of intellectual and ethical development de¬ 
scribed students’ progression through a series of developmental stages; in 
addition, he identified several ways in which students may delay, retreat 
or escape from this intellectual progression. Kolb (1984) outlined an ex¬ 
perimental learning cycle model, in which students’ learning and cognitive 
growth matures through repeated cycles of direct experience, reflection, 
conceptualization and subsequent application. Baxter-Magolda’s (1992) 
epistemological reflection model emphasized gender-based differences in 
patterns of reasoning and learning that emerge over time. A report by the 
American Association of Colleges and Universities (AAC&U) (2002) ad- 
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vocated cultivation of intentional learners, using many of the elements de¬ 
scribed by Kolb (1984), and stressed the need for locally-devised, authen¬ 
tic assessments that emphasize evidence of higher-level learning. 

Evident in these and other models of intellectual development 
is the recognition that student learning is cumulative and multifactorial; 
moreover, they imply that the impact of individual experiences may be 
limited or at least difficult to assess if (a) students are not developmentally 
ready to assimilate the experience, (b) the experience is viewed outside a 
progressive context, or (c) the experience or assessment thereof does not 
align with students’ particular ways of knowing. By integrating a large 
number of information-rich variables into a comprehensive assessment of 
students’ total university experience, it is possible to reflect the cumulative 
nature of learning while accommodating potentially confounding effects 
of cognitive stage, readiness to learn, and delivery modality. 

Here we utilize a multivariate approach to describe and quantify 
growth of a student engagement culture on our campus. Using data from a 
self-report survey of upperclass students, we address the following ques¬ 
tions: (a) What are the discrete factors of engagement that characterize the 
university experience at our institution; (b) To what extent have students’ 
perceptions of their engagement experiences changed over time; and (c) 
How are the observed patterns of change related to institutional priorities? 
We discuss our findings with respect to the need for institution-level as¬ 
sessment of learning and the student experience that is consistent, quanti¬ 
tative and comprehensive. 


Methods 


Institutional Context 

Western Kentucky University (WKU) is a comprehensive public 
university with a total enrollment just above 20,000, of which approximate¬ 
ly 80 % are undergraduates. WKU is accredited by the Southern Association 
of Colleges and Schools (SACS), and last underwent reaffirmation of ac¬ 
creditation in 2005. As part of the reaffirmation process, SACS institutions 
are charged with developing and implementing a Quality Enhancement 
Plan (QEP) that reflects key issues arising from ongoing institutional as¬ 
sessment processes and engages a broad base of institutional constituencies 
in addressing one or more components of the university’s student learning 
mission (Commission on Colleges, 2008). In particular, QEPs are expect¬ 
ed to identify discrete student learning outcomes, action steps, institutional 
resources and assessments tied to an overarching student learning theme. 
WKU’s plan, Engaging Students for Success in a Global Society, employs 
a student engagement approach to enhancing students’ (a) capacity to apply 
their knowledge and training to address societal issues; (b) appreciation for 
the diversity of peoples, ideas and cultures; and (c) awareness of their op- 
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portunities and responsibilities as citizens in a global society (WKU, 2005). 
Implementation of the plan began in Fall 2005. 

Survey Instrument 

Since 2001, WKU has utilized the WKU Student Engagement 
Survey (WKUSES) as a means of assessing various aspects of students’ 
WKU Experience, particularly curricular and co-curricular opportunities 
for engaged learning; a number of the items on this survey are similar to 
those included in the NSSE survey. WKUSES is administered each spring 
to juniors and seniors through students’ major departments; students com¬ 
pleting associate’s degree programs also have the opportunity to complete 
the survey. There exist a core set of university-wide items (see Appendix) 
that focus on the extent to which students have been involved in various 
types of experiential learning opportunities, the degree to which aspects of 
their coursework have contributed to essential knowledge, skills and per¬ 
spectives, and the academic relationships among peers and faculty men¬ 
tors. The number of respondents is typically 2500-3000 per year. Partici¬ 
pation is voluntary and anonymous. Colleges and departments have the 
option of including additional questions to be asked of their students. Tra¬ 
ditionally, item-by-item results of the survey have been distributed to col¬ 
leges and departments for their formative use. 

For this study, we utilized a set of 45 university core items (in¬ 
put variables) common through the 2005-2008 administrations. Each item 
had a 4-6 point response scale; questions used in the analysis, and accom¬ 
panying summary statistics, appear in Table 1. 

Table 1 


WKUSES Variables, Response Scales and Summary Statistics 


Input variables 

QEP Scale 

N 

2005 Mean±S£' 

Thinking about the advising you received, 
rate the following: 

Overall quality of advisor 

4 

2639 

2.909±0.018 

Availability of advisor 

4 

2628 

2.963±0.017 

Advisor’s help in developing your schedule 

4 

2584 

2.837±0.019 

each semester 

Advisor’s help with career planning 

4 

2435 

2.429±0.021 

Advisor’s knowledge of degree requirements 

4 

2623 

3.229±0.017 

To what extent has course work in your major 
contributed to your: 

Acquiring job/work-related knowledge? 

5 

2568 

3.727±0.019 

Writing clearly and effectively? 

5 

2657 

3.578±0.019 

Speaking clearly and effectively? 

5 

2656 

3.586±0.019 

Thinking critically and analytically? 
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2656 

3.994±0.017 
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Table 1 (continued) 


Input variables 

QEP 

Scale 

N 

2005 Mean±S£ 

Applying theories/concepts to practical 
problems/new situations? 


5 

2655 

3.891±0.017 

Working effectively with others? 

Y 

5 

2664 

3.88740.018 

Enthusiasm towards this discipline? 


5 

2644 

3.87640.019 

Use of the library? 


5 

2640 

2.91240.022 

Use of computer technology? 

To what extent have general education courses 
contributed to your: 


5 

2661 

3.78540.020 

Acquiring a broad general education? 

Y 

5 

2650 

3.38240.019 

Writing clearly and effectively? 


5 

2648 

3.32940.018 

Speaking clearly and effectively? 


5 

2642 

3.28240.019 

Thinking critically and analytically? 


5 

2643 

3.28440.019 

Understanding diverse cultures? 

To what extent did University Experience 
contribute to: 

Y 

5 

2640 

3.25840.020 

Your college experience? 


5 

2099 

1.98940.024 

Involving you in campus and community? 

As a student, how often have you: 

Y 

5 

2102 

1.91640.023 

Discussed career plans with a faculty member? 


5 

2684 

2.95640.020 

Been challenged to meet the expectations 
of faculty? 


5 

2674 

3.56340.019 

Discussed ideas from your classes with 
others outside class? 

Y 

5 

2674 

3.73540.019 

Participated in practicum courses, 
internships, co-ops, clinical assignments 
or field experiences? 

Y 

5 

2650 

2.88540.027 

Interacted with people from different 
backgrounds or cultures? 

Y 

5 

2656 

3.45740.020 

Performed community service/volunteer 
activities? 

Y 

5 

2658 

2.72140.024 

Participated in research projects with a 
faculty member? 

Y 

5 

2651 

1.78040.022 

Presented a project or paper with a faculty 
member? 

Y 

5 

2649 

1.64840.021 

Attended a play, concert, art exhibit, lecture, 
or cultural event? 

Y 

5 

2670 

2.92340.021 

Participated in meetings/activities of a 
professional organization? 

Y 

5 

2656 

2.67940.025 

Participated in study abroad or other 
international activities? 

Y 

5 

2648 

1.47440.019 

Participated in leadership training or had a 
leadership role? 

Y 

5 

2653 

2.29840.027 

Attended a university athletic event? 


5 

2671 

2.93640.027 

Worked with a faculty member on a project 
outside of class? 

Y 

5 

2669 

1.77240.022 

Been contacted by your advisor regarding 
your degree program? 


5 

2671 

2.02840.021 

(continued) 
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Table 1 (continued) 


Input variables 

QEP 

Scale 

N 

2005 Mean ±SE 

While at the university, on average, how many 
hours per week have you spent working in a job? 


8 

2667 

4.880±0.044 

How many hours per week do you spend doing 
volunteer work? 

Y 

5 

2670 

1.949±0.022 

How many courses in your major included 
knowledge or activities that will help you 
become a contributing citizen? 

Y 

5 

2650 

3.217±0.020 

How many of your non-major courses included 
knowledge or activities that will help you 
become a contributing citizen? 

Y 

5 

2638 

2.834±0.019 

How many of the courses in your major 
required community involvement, volunteerism, 
or other out-of-class learning activities? 

Y 

5 

2664 

2.534±0.023 

How many of your non-major courses required 
community involvement, volunteerism, or other 
out-of-class learning activities? 

Y 

5 

2651 

2.143±0.018 

During the past year, how often have you 
participated in a community-based project as 
part of your homework? 

To what extent has your coursework: 

Y 

4 

2660 

1.566±0.015 

Contributed to your voting in local, state, 
or national elections? 

Y 

4 

2660 

1632±0.017 

Affected your contribution to the welfare 
of your community? 

Y 

4 

2655 

1.611±0.015 

QEP outcome variables 


Scale 1 

N 

2006 Mean±S£ 


To what extent do you feel you: 


Have an understanding of the major issues facing 
society today? 

4 

2401 

2.811±0.014 

Can apply what you have learned in your major 
to resolving problems or concerns in society? 

4 

2394 

2.794±0.015 

Have the experience needed to appreciate both 
sides of an issue? 

4 

2386 

3.086±0.015 

Are aware of your responsibilities as a citizen, 
living and working in a global society? 

4 

2398 

2.950±0.015 


Note. Values presented for input variables are derived from the Spring 2005 administra¬ 
tion; these 45 variables represent those used in the factor analysis. Values for QEP outcome 
variables are derived from the Spring 2006 administration (the first year these items were 
included). Input variables related to QEP priorities are indicated. Response scales for each 
item are as follows (variables are denoted as Input 1 45 and Outcome 1 —4, based on the 
order in which they appear in the table): 

Input 1-5: 1 = Poor, 2 = Fair, 3 = Good, 4 = Excellent 

Input 6-21: 1 = Not at all, 2 = Very little, 3 = Some, 4 = Quite a bit, 5 = Very much 

Input 22-32: 1 = Never, 2 = Rarely, 3 = Occasionally, 4 = Often, 5 = Very often 
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Input 33: 1 = 0,2 = 1-5,3 = 6-10,4=11-15,5 = 16-20, 6 = 21-25,7 = 26-30,8 = 30+ 

Input 34: 1 = 0, 2 = 1-2, 3 = 3-4, 4 = 5-9, 5 = 10+ 

Input 35-38: 1 = Never, 2 = Rarely, 3 = Occasionally, 4 = Often, 5 = Very often 

Input 39-42: 1 = None, 2 = Very few, 3 = Some, 4 = Quite a few, 5 = Almost all 
Input 43: 1 = Never, 2 = Sometimes, 3 = Often, 4 = Very often 

Input 44-45: 1 = Very little, 2 = Some, 3 = Quite a bit, 4 = Very much 

Outcome 1-4: 1 = No extent, 2 = Little extent, 3 = Some extent, 4= Great extent 

Values of skewness and kurtosis calculated for each item gener¬ 
ally were between -1.00 and +1.00, a range typically regarded as evidence 
of normality (Kuh, 2001b). Items showing higher degrees of skewness (all 
positive) tended to be those associated with the extent of involvement in 
out-of-class or optional experiential activities, including independent re¬ 
search, study abroad, service learning, and volunteerism (Table 1); such 
patterns are also to be expected (Kuh, 2001b). 

Beginning in 2006, four items were added to the WKUSES sur¬ 
vey. These questions were designed to provide information on the extent 
to which students felt their overall university experience had contributed 
to outcomes specifically targeted in our QEP; Table 1 provides definitions 
and summary statistics for these QEP outcome variables. 

Data Analysis 

All statistical analyses were conducted using SYSTAT 11 for 
Windows (SYSTAT Software, 2004). The Spring 2005 administration of 
WKUSES was taken as a baseline year, as it pre-dated implementation of 
our QEP. Spring 2005 data for the 45 input variables were subjected to prin¬ 
cipal components analysis (PCA) of the correlation matrix derived from 
the original data. Principal components analysis was chosen over common 
factor analysis, as ordinations derived from the former incorporate both 
communality as well as uniqueness (Tan, 1992); moreover, the principal 
components algorithm does not limit the number of factors that can be ex¬ 
tracted from the data. The correlation matrix results in each original vari¬ 
able contributing an equal amount of total variation to the data set; this en¬ 
sures that the solution is not dominated by those variables with the highest 
variance, as could occur if the variance-covariance matrix were used. 

Resulting factors were orthogonally rotated using the Varimax 
procedure. Varimax tends to spread the percent of variance explained more 
evenly across factors, while also polarizing the loadings of individual vari¬ 
ables on each factor (resulting in a few variables with high loadings on a 
given factor, and the remainder with loadings near zero); this simplifies in¬ 
terpretation of each factor. Component loadings of individual variables on 
rotated factors were used to categorize the derived axes. 

The 2005 raw data were projected onto the space defined by the 
rotated factors to generate a baseline data set of scores for comparison 
over time. By definition, mean factor scores for all axes in 2005 were 0.00. 
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Data from 2006, 2007, and 2008 administrations of WKUSES were simi¬ 
larly projected onto the space defined by the 2005 factors. 

Differences in 2006, 2007 and 2008 mean values for both individ¬ 
ual input variables and factor scores were tested for significant differences 
relative to 2005 baseline values using ANOVA, with significance based on 
Bonferroni-adjusted criteria. QEP outcome variables were also tested for 
significant differences among years (2006 through 2008) using ANOVA; 
here all pairwise comparisons were tested for significance using Tukey’s 
HSD multiple comparison test. 


Results 


Identification of Factors 

The principal component analysis generated ten orthogonal fac¬ 
tors with eigenvalues greater than 1.0; these were taken to be factors con¬ 
taining significant information. These 10 principal components accounted 
for 61.34% of the total variation in the data set. The first principal compo¬ 
nent accounted for 21.39% of the total variation; the remaining nine fac¬ 
tors accounted for between 8.15 and 2.35% of the variation. 

The rotated factors also accounted for 61.34% of the total varia¬ 
tion; however, no single factor accounted for more than 10.06 % of the to¬ 
tal variation. The percent of variance explained by each rotated factor is 
given in Table 2, sorted by percent of total variance explained. For each 
rotated factor (with one exception—see Factor X), loadings of important 
variables all had the same sign, indicating that students responding posi¬ 
tively for one tended to do so for all. 

Table 2 

Summary of 10 Rotated Factors Derived from Analysis of the 2005 
WKUSES Data 


% 

Factor Variance Contributing variables 


Factor 

Loadings classification 


I 10.06 Major course impact on speaking skills 
Major course impact on thinking skills 
Major course impact on writing skills 
Major course impact on applying concepts 
Major course impact on enthusiasm for 
discipline 

Major course impact on ability to work 
with others 


0.735 Major course 

0.726 meta-skills 

0.720 

0.706 

0.672 

0.649 


(continued) 
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Table 2 (continued) 



% 



Factor 

Factor 

Variance 

Contributing variables 

Loadings 

classification 

II 

8.52 

General education impact on writing skills 

0.847 

General 



General education impact on thinking skills 

0.844 

education 



General education impact on speaking skills 

0.840 

(QEP) 



General education contribution to broad 

0.790 




education 

Contribution to understanding diverse cultures 

0.747 


III 

8.38 

Overall quality of advising 

0.867 

Advising 



Advisor’s helpfulness in developing schedules 

0.866 




Advisor’s helpfulness with career planning 

0.786 




Availability of advisor 

0.782 




Advisor’s knowledge of degree requirements 

0.764 


IV 

6.94 

Presentation of project or paper with faculty 

0.802 

Experiential 



Participation in research projects with faculty 

0.796 

activities 



Work with faculty on a project outside of class 

0.731 

(QEP) 



Participation in study abroad/ 

0.658 




internationalization 



V 

6.19 

Participation in community service/ 

0.758 

Community 



volunteerism 


involvement 



Hours per week doing volunteer work 

0.660 

(QEP) 



Participation in activities of professional 

0.589 




groups 

Participation in leadership training programs 

0.563 


VI 

5.52 

Discussion of class ideas outside of class 

0.722 

Academic 



Extent of challenge by faculty expectations 

0.699 

climate 



Discussion of career plans with faculty 

0.612 


VII 

4.90 

Community involvement in major courses 

0.701 

Curricular 



Community involvement in non-major 

0.660 

citizenship 



courses 


(QEP) 



Major course contribution to citizenship 

0.615 




potential 



VIII 

4.02 

Course role in college experience 

0.907 

University 



Course role in campus/community 

0.890 

Experience 



involvement 



IX 

3.75 

Coursework impact on voting 

0.857 

Civic 



Coursework impact on civic commitment 

0.714 

behavior 

(QEP) 


(continued) 
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Table 2 (continued) 


% 



Factor 

Factor Variance 

Contributing variables 

Loadings 

classification 

X 3.03 

Attendance at university athletic events 

0.628 

Cultural 


Number of hours worked per week 

-0.529 

engagement 


Note. Factors are sorted by the percent of variance explained. Contributing variables are 
defined as those input variables showing high loadings (positive or negative) on a given 
factor. Factors directly related to QEP priorities are indicated in the factor classification. 


Supplementary analyses supported the stability and reliability of 
the derived factors. Data from the 2003 and 2004 administrations gen¬ 
erated nearly identical eigenvalues and patterns of factor loadings when 
subjected to equivalent analyses (McElroy, 2006). Reliability studies (J. 
Bruni, personal communication, 2006) indicated that factors were robust 
to removal of individual variables from the analysis. Patterns of loadings 
(Table 2) indicated that sets of related items tended to load on the same 
axes, suggesting that survey questions were being appropriately interpret¬ 
ed by students and measuring what they were intended to measure. 

The first rotated factor (Factor I) explained 10.06% of the total 
variation. Variables showing high loadings on this factor were those that 
focused on the contribution of major program coursework to development 
of students’ speaking, critical thinking, and writing skills, as well as their 
ability to apply concepts and theories, feel enthusiastic towards their dis¬ 
cipline and work effectively with others. Factor I was thus categorized as 
a Major Program Meta-Skills axis. 

Factor II explained 8.52% of the total variation. Factor II was clas¬ 
sified as a General Education axis. Variables addressing the value of gen¬ 
eral education in promoting writing, critical thinking, and speaking skills, 
as well as development of a broad education and understanding of diverse 
cultures, all had high loadings on this axis. 

Factor III explained 8.38% of the total variation and represented 
an Advising axis. Variables with high loadings included questions reflect¬ 
ing the overall quality of advising, advisors’ helpfulness in developing the 
academic schedule and career plans, and advisors’ availability and knowl¬ 
edge of degree requirements. 

Factor IV accounted for 6.94% of the total variation. Variables 
reflecting the students’ presentation of research results done under the di¬ 
rection of a faculty member, the extent of students’ participation in re¬ 
search or other projects with faculty, as well as their involvement in study 
abroad or other international activities loaded heavily on this axis. Factor 
IV could thus be broadly classified as an Experiential Activities axis. 

Factor V accounted for 6.19% of the total variation. Factor V was 
categorized as a Community Involvement axis, reflecting the extent of stu- 
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dents’ participation in community service or volunteerism activities. To a 
lesser extent, this axis also reflected the extent of students’ involvement in 
meetings of a professional organization and leadership training programs. 

Factor VI explained 5.52% of the total variation. Variables show¬ 
ing high loadings on this axis included those focusing on the extent to 
which students discussed class ideas with others outside of class, felt chal¬ 
lenged by faculty expectations, and discussed career plans with faculty. As 
such, Factor VI was broadly classified as an Academic Climate axis. 

Factor VII accounted for 4.90% of the total variation. Variables 
with high loadings on this axis addressed the extent to which students’ ma¬ 
jor and non-major courses required community involvement as an element 
of the course or advanced their capacity to be a contributing citizen. Factor 
VII was thus defined as a Curricular Citizenship axis. 

Factor VIII explained 4.02% of the total variation. Factor VIII 
represented a University Experience axis. Variables important to this axis 
were those that reflected the extent to which WKU’s course for first-se¬ 
mester students contributed to their college experience and their involve¬ 
ment with campus and community. 

Factor IX explained 3.75% of the total variation. Variables with 
high loadings on this axis included those addressing the extent to which 
students’ coursework affected their tendency to vote in elections and con¬ 
tribute to community welfare. While similar to Factor VII, this axis fo¬ 
cused more strongly on the extent to which coursework impacted students’ 
civic behavior, whereas Factor VII reflected the incorporation of socially- 
relevant topics or experiences into the curriculum. Thus, Factor IX was de¬ 
scribed as a Civic Behavior axis. 

Factor X accounted for 3.03% of the total variation. Factor X was 
driven by responses to the question of how often students attended a uni¬ 
versity athletic event, and to a lesser extent other cultural events, con¬ 
trasted against the number of hours per week students worked in a job. As 
such, Factor X constituted a Cultural Engagement axis. The fact that at¬ 
tendance items were opposite in sign from the hours worked variable indi¬ 
cates that the more hours a student worked per week, the less they tended 
to participate in the types of activities reflected on this axis. 
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Advisor’s help in developing schedule 2.837±0.019 2.959±0.019* 

Advisor’s help with career planning 2.429±0.021 2.595±0.021* 

Major courses: Speaking skills 3.586±0.019 3.721±0.019* 

Major course: Applying concepts/theories 3.891±0.017 4.016±0.017* 











Input variable 2005 Mean±£E 2006 Meani.SE 2007 MeaniSE 2008 MeaniSE 

Discussion of career plans with faculty 2.956±0.020 3.136±0.021* 
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Demonstrating Growth of a Student Engagement Culture 


Change in Input Variables and Factor Scores Over Time 

Eight of 45 input variables (17.8%) showed significant differenc¬ 
es in means between 2005 and 2006, based on Bonferroni-adjusted crite¬ 
ria {p < .001). The number of significant comparisons rose to 13 out of 45 
by 2007 (28.9%), and 23 out of 45 variables (51.1%) showed significant 
differences between 2005 and 2008. All statistical differences represented 
increases in mean values in 2006 relative to 2005. Overall, only 7 of 135 
total pairwise comparisons showed decreases in means over time; of these, 
only two were statistically significant, both involving the same input vari¬ 
able. A summary of the statistical results is provided in Table 3. 

Of the 45 input variables, 23 directly reflect QEP priorities (Table 
1). Of these, five (21.7%) showed significant differences in 2006 relative 
to 2005; this number increased to 10 of 23 in 2007 (43.5%) and 14 of 22 
(60.9%) in 2008 (Table 3). 

Two of 10 factors (20%) showed significant differences inmean score 
between 2005 and 2006, based on Boneferroni-adjusted criteria (p < .005). 
The number of significant comparisons increased to four of 10 (40%) by 
2007. In 2008, four of 10 factors (40 %) also showed significant differ¬ 
ences relative to 2005 means; however, these differences involved a some¬ 
what different set of factors (Table 3). Five of the 10 factors directly reflect 
QEP priorities (Table 2). Of these, 1 (20%) showed a significant difference 
in 2006 relative to 2005. This number increased to three of five (60%) by 
2007; in 2008, only two of five (40%) were significant at the Bonferroni 
level, though a third approached significance {p = .025) (Table 3). 

Change in QEP Outcome Variables Over Time 

Means of each of the four QEP outcome variables increased sig¬ 
nificantly within one year, from 2006 to 2007 (Figure 1). All four variables 
showed additional increases in 2008; two of these (appreciating both sides 
of an issue, awareness of responsibilities as a global citizen) showed sig¬ 
nificant differences between 2007 and 2008. Associated z-scores indicat¬ 
ed that these variable means increased 20.00 to 28.56 standard deviations 
from 2006 to 2008; by way of comparison, a change of approximately 2 
standard deviation units would be considered statistically significant. 
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Figure 1. Significant increases in QEP outcome variables over time. Outcome variables 
asked respondents “To what extent do you feel you: a) have an understanding of the major 
issues facing society today; b) can apply what you have learned in your major to resolving 
problems or concerns in society; c) have the experience necessary to appreciate both sides 
of an issue; and d) are aware of your responsibilities as a citizen, living and working in a 
global society ?” Each of the four variables increased significantly relative to the baseline 
year of 2006 based on the results of Tukey’s HSD multiple comparison test; for variables 
shown in c) and d), all pairwise comparisons were significant. Between 2006 and 2008, 
outcome variables increased by a) 21.43, b) 20.00, c) 20.58, and d) 28.46 standard devia¬ 
tion units. Standard error bars were too small to be depicted on the graphs, but ranged from 
0.012 to 0.015. 


Discussion 

Student Self-Reports Indicate Growth of a Student Engagement Culture 
on our Campus 

Data from our institution indicate that student perceptions of their 
university experience have changed significantly over a short period of time. 
In most cases, these changes are consistent with growing areas of empha- 
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sis at the university, centered on student engagement. We also see evidence 
of ancillary benefits not specifically targeted by programmatic initiatives, as 
well as the impact of learning opportunities provided by world events. 

In recent years, the university has identified engaged learning, 
community engagement and internationalization as key priorities with¬ 
in the academic mission. Involving students in independent research and 
scholarly activity has been a long-standing emphasis in a number of disci¬ 
plines (particularly the natural and behavioral sciences, but now expressed 
more broadly). Responding to NSSE data, student satisfaction with advis¬ 
ing has also been a target for improvement across our campus. It appears 
that students perceive the impact of these priorities and various curricular 
and co-curricular initiatives designed to address them. Our data indicate 
significant increases in mean scores for the Experiential Activities, Curric¬ 
ular Citizenship and Advising factors (Table 3); these factors are heavily 
influenced by input variables directly related to the priority areas identi¬ 
fied above (Table 2). 

QEP outcome variables showed the greatest positive change dur¬ 
ing the study period (Figure 1). These items asked students to contextual¬ 
ize their overall university experience in four dimensions consistent with 
our QEP targeted student learning outcomes. Students reported increas¬ 
ingly greater awareness of key societal issues, enhanced capacity to ap¬ 
ply their knowledge and skills in addressing societal concerns, increased 
appreciation of diverse perspectives, and a greater sense of social respon¬ 
sibility and accountability. These enhanced outcomes are what we would 
hope to see, given that students also reported greater involvement in ex¬ 
periential and community-based activities as part of their university ex¬ 
perience. While we do not establish a causal relationship among process 
and outcome variables, the fact that both sets are increasing in a correlat¬ 
ed fashion is reassuring, and supports the oft-held view that integration of 
educational best practices should result in at least perceived gains in stu¬ 
dents’ knowledge, skills and perspectives (Kuh, 2001a, 2001b; Mehaffy, 
2005; Colby et al., 2003, 2007). 

Interestingly, the Academic Climate factor also showed signifi¬ 
cant positive change during the course of our study. This factor was in¬ 
fluenced by variables associated with the degree of interaction and expec¬ 
tations among students and faculty outside the classroom. Such cultural 
enhancement was neither anticipated nor specifically targeted; neverthe¬ 
less, it may have resulted from changes in pedagogy instituted with more 
discrete outcomes in mind, such as community-based projects and/or in¬ 
dependent scholarship. 

Two variables in our analysis showed significant decreases in 
mean score over time. In 2006 and 2007, respondents felt that, on average, 
their coursework contributed less to their voting in elections than was felt 
by respondents in 2005. Similarly, the Civic Behavior factor, which was 
heavily influenced by this input variable, also decreased in 2006 and 2007 
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relative to 2005. These decreases were significant in 2007 relative to 2005. 
While such results initially suggest a declining emphasis on civic and po¬ 
litical engagement in our courses, this is not necessarily the best way to 
view these data. It is important to note that the Spring 2005 administration 
followed close behind the 2004 presidential election, during which time 
many campuses (including our own) leveraged interest in the election as a 
teaching moment, devoting considerable attention to the political dimen¬ 
sions of issues across the disciplines. 

Studies on student self-reports (Converse & Presser, 1989; Kuh, 
2001b) have indicated that responses are more accurate when students are 
asked to reflect on relatively recent activities; by extension, this suggests that 
recent experiences contribute somewhat more to students’ perceptions than 
do activities that occurred earlier (or did not occur as frequently) in their uni¬ 
versity careers. As a result, the 2005 baseline value may have been some¬ 
what inflated, reflecting heavy emphasis on political aspects of course topics 
during the previous semester. The fact that mean values for these same items 
declined for two years before beginning to converge on 2005 values dur¬ 
ing the 2008 administration, as interest and media attention on the upcom¬ 
ing 2008 elections grew, supports this interpretation. We have observed the 
same cyclical pattern in NSSE data for our institution, and other campuses 
appear to have experienced a similar effect (J. Hayek, personal communica¬ 
tion). As a result, it may be problematic to rely on short-term assessments of 
civic or political engagement, given that increased curricular focus tied to 
national elections may well drive responses in items such as these. 

An Intentional Focus on Engagement Can Lead to Rapid Change 

While most univariate and multivariate indicators showed positive 
trends throughout the course of our study, those related to our QEP focus 
tended to show the largest and most rapid gains. The proportion of QEP- 
related input variables and factors showing significant positive change in¬ 
creased faster than non-QEP aspects of engagement and the overall univer¬ 
sity experience. This suggests that an intentional focus on clearly articulated 
goals can be a powerful catalyst in promoting change in student learning 
indicators, even when the linkage between programming and assessment 
is indirect. 

Our QEP has the stated goal of “engaging students with commu¬ 
nities other than their own in purposeful learning activities that explicitly 
address their capacity and responsibility to contribute to society in posi¬ 
tive ways” (Western Kentucky University, 2005, p. 3). This goal is tied 
to student learning outcomes that specifically seek to engage students as 
practitioners of their disciplines, develop their awareness of diverse per¬ 
spectives, and cultivate their sense of social responsibility. While the goal 
and outcomes are clearly articulated, departments and programs have the 
freedom to address these outcomes in ways that are relevant to their stu- 
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dents and their disciplines. Our intended avoidance of a “one size fits all” 
approach to implementation means that there is not a simple articulation 
between implementation of curricular or co-curricular initiatives and the 
assessment of their impact on student learning at the institutional level. 
Nevertheless, our results show that this does not impede progress toward 
meeting institutional objectives; in fact, it may facilitate rapid change, by 
encouraging innovative programming that may be difficult to assess direct¬ 
ly, especially if a common measure or rubric is expected to be applied. 

The factors displaying significant change are not necessarily among 
those accounting for the highest proportions of total variation in the data 
set; the Major Course and General Education Meta-Skills factors alone rep¬ 
resented nearly a third of the variation explained in our factor model, yet 
scores on these axes did not change significantly during the study (though 
the Major Courses Meta-Skills factor approached significance). Moreover, 
the input variables loading heavily on the Experiential Activities, Curricu¬ 
lar Citizenship and Civic Behavior factors had, as might be expected (Kuh, 
2001b), some of the lowest means seen in the survey (Table 1, Table 2). 
Nevertheless, these factors were among the most responsive. 

Most would probably agree that commitment to enhancing stu¬ 
dents’ communication, critical thinking, and problem solving skills is a 
fundamental element of all universities’ academic missions, and at least 
as important as advancing engagement-related goals; certainly this is the 
case on our campus. Given this ubiquity and emphasis, why then do we 
not see equivalent change in the Meta-Skills factors as in more experien- 
tially-driven factors (even though some of the variables contributing to 
these factors are QEP-related)? We suggest again that clear articulation 
of goals and outcomes through our QEP process has provided the inten- 
tionality and focus necessary to effect demonstrable change along student 
engagement vectors; had our QEP focused directly on communication or 
critical thinking, we would expect to have seen significant change mani¬ 
fest in those dimensions to the partial exclusion of others. 

A Multivariate Approach Provides a Powerful, Hypothesis-Driven Para¬ 
digm for Assessment 

Our study presents a model for institution-level assessment of stu¬ 
dent learning goals and outcomes. It accommodates alternative, interrelat¬ 
ed approaches to addressing common objectives and better reflects the cu¬ 
mulative, multifactorial nature of student learning. Moreover, it provides 
a statistically robust mechanism for assessing development of students’ 
higher-order knowledge, skills and perspectives. 

Tan (1992) identified three limitations of a univariate approach to 
assessment, namely (a) the inability to apply a common yardstick across di¬ 
verse programs, (b) the potential for short-term annual fluctuations in indi¬ 
cators that may mask underlying patterns, and (c) lack of information avail- 
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able about the interrelationships of variables and their impacts on targeted 
outcomes. We agree, and also suggest that highly structured, univariate in¬ 
dicators make a priori assumptions about relationships that constrain the re¬ 
sulting analysis and may preclude discovery of unanticipated correlations. 

By collecting data on a large array of potential indicators, and sub¬ 
jecting these to a multivariate analysis, it is possible to test the viability 
of predicted relationships while retaining the potential to uncover novel 
interactions. For example, we would not have predicted emergence and 
strength of our Academic Climate factor. In addition, redundant indicators 
can be developed and included so as to accommodate diverse approaches 
to addressing the same targeted outcome without compromising statistical 
validity. Our Experiential Activities factor draws on correlated patterns of 
variance among indicators of independent scholarly research, faculty-di¬ 
rected projects and study abroad, all of which are means of engaging stu¬ 
dents with real-world situations outside the traditional classroom expe¬ 
rience. Finally, the existence of redundancy minimizes the potential for 
chance fluctuations in one or two variables that might dominate the results 
of the analysis. The input variable assessing major programs’ contribu¬ 
tion to students’ use of computer technology showed an inconsistent pat¬ 
tern of univariate statistical significance among years (Table 3); however, 
this variable did not load heavily on any factor, and so was not subject to 
overinterpretation. 

Our emphasis on assessing students’ overall university experience 
aligns well with cumulative, multifactorial models of learning (AAC&U, 
2002; Baxter-Magolda, 1992; Kolb, 1984; Perry, 1970). It minimizes the 
possibility that the learning value of a single, well-designed experience 
may not be recognized by the student or otherwise demonstrable in the 
short-term. Moreover, by designing items which ask students to reflect on 
the extent to which certain types of related experiences contributed to their 
university experience, our approach targets accumulated learning gains 
and complementary modes of delivery/leaming. 

Implications for Practice 

From an institutional perspective, it is preferable to have a limited 
number of inclusive measures of student learning and success; however, 
this is often difficult, given the diversity of programs, missions and priori¬ 
ties of component units. We have chosen to identify a set of overarching 
indicators that reflect different dimensions of our academic priorities, yet 
leave open the manner by which programs address targeted outcomes. In 
this way, the university can derive a comprehensive yet concise picture 
of its teaching and learning efficacy, without becoming bogged down in a 
level of detail that may cloud or impede strategic, formative thinking. 

How might such data be used to further institutional goals and 
promote innovative thinking about and application of high-impact prac- 
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tices that advance academic quality? First, by tracking change among a 
finite set of multivariate indicators, an institution can more objectively 
and fairly assess the progress of units relative to one another; these trends 
can be used to identify best practices among a set of diverse strategies, 
guide allocation of resources, and reinforce a common understanding and 
commitment to advancing university priorities. Second, academic depart¬ 
ments and programs can be free to pursue a range of initiatives (within or 
among units) that each contribute to the same underlying outcome, but in 
ways that make sense for their particular mission and students; pedagogy, 
not ease of assessment, can drive units’ curricular and co-curricular deci¬ 
sion-making. Finally, institutions can use multivariate, endpoint data to 
reinforce integrative, interdisciplinary, outcomes-based thinking that pro¬ 
motes linkages between Academic Affairs and Student Affairs or among 
academic disciplines and the community beyond the bounds of campus. 

This is not to say that such an approach represents the end-all of 
demonstrating effectiveness in teaching and learning. Student self-reports 
represent but one dimension of the assessment process. It is important for 
academic programs to design student learning indicators that are more pre¬ 
cise in defining causal relationships between program elements and learn¬ 
ing; only in this way can departments maintain curricula that continue to 
be impactful, responsive and relevant. Nevertheless, at the institutional 
level it may well be sufficient to adopt a broader view and ask “Have our 
graduates achieved the essential learning outcomes we as an institution 
had hoped and set forth for them?” 
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Answer each of the survey questions in light of your experiences while attending WKU. 


1. Thinking about the ADVISING you received in your major, rate the following: 


2 . 



Poor 

Fair 

Good 

Excellent 

Don't Know 

a overall quality of advising. 

.O 

o 

o 

O 

o 

b availability of advisor. 

.o 

o 

o 

o 

o 

c. advisor's help with developing your schedule each semester 

. n 

o 

o 

o 

o 

d advisors's help with career planning. 

. o 

o 

o 

o 

o 

e. advisor's knowledge of degree requirements. 

. n 

o 

o 

o 

o 


To what extent has the COURSE WORK in your MAJOR contributed to your: 

Not at All Very Little Some 

Quite a Bit 

Very Much 

N/A 

Don't Know 

a. acquiring job/work-related knowledge?. 

.O 

o 

o 

o 

o 

O 

b. writing clearly and effectively?. 

.O 

o 

o 

o 

o 

o 

c. speaking clearly and effectively?. 

.O 

o 

o 

o 

o 

o 

d. thinking critically and analytically?. 

.O 

o 

o 

o 

o 

o 

e. applying theories/concepts to practical 

problems/new situations?. 

.O 

o 

o 

o 

o 

o 

f working effectively with others?. 

. n 

o 

o 

o 

o 

o 

g enthusiasm toward this discipline 9 . 

.o 

o 

o 

o 

o 

o 

h use of the library 9 . 

. o 

o 

o 

o 

o 

o 

i. use of computer technology 9 

.o 

o 

o 

o 

o 

o 


To what extent have your GENERAL 

EDUCATION COURSES contributed to your: 


N/A 

Not at All 

Very Little 

Some 

Quite a Bit 

Very Much 

Don't Know 

a acquiring a broad general education 9 .. 

. o 

o 

o 

O 

o 

o 

b. writing clearly and effectively 9 . 

. o 

o 

o 

O 

o 

o 

c. speaking clearly and effectively?. 

. o 

o 

o 

O 

o 

o 

d thinking critically and analytically?. 

. o 

o 

o 

O 

o 

o 

e understanding diverse cultures? 

. o 

o 

o 

O 

o 

o 


4. To what extent: 


No 

Extent 


Little 

Extent 


Some 

Extent 


Great 

Extent 


5. 


a. do you have an understanding of the major issues facing society today?. 

b. can you apply what you have learned in your major to resolving 

problems or concerns in society?. 

c. do you have the experience necessary to appreciate both sides of an issue?... 

d. are you aware of your responsibilities as a citizen, living and working 

in a global society?. 

e are empowered to continue learning throughout life? . 

f. were computer labs at WKU adequate to meet your learning needs?. 

g were online learning systems at WKU adequate to meet your learning needs? 


To what extent did University Experience (Freshmen Seminar) contribute 

Not at All Very Little 


a. your college experience?.O 

b. involving you in the campus and community?.O 


o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


to: 



N/A 

Some 

Quite a Bit 

Very Much 

Don't Know 

o 

O 

O 

o 

o 

O 

O 

o 


6. Has your time to graduation increased because courses for your major were not offered during terms 
when you and your advisor had planned for you to take them? 

O Yes O No 
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® CD © © (D © © © GD (D 
®©©®©©©©®® 
®©©®®©©®®® 
®©©©©©©©©© 
®©©©©©©©©© 


As a student at WKU, HOW OFTEN have you: 

Never 

Rarely 

Occasionally 

Often 

Very Often 

N/A 

Don't Know 

a. discussed career plans with a faculty member? . 

... o 

o 

o 

o 

o 

o 

b. been challenged to meet the expectations of faculty?. 

... o 

o 

o 

o 

o 

o 

c. discussed ideas from your classes with others outside class?. 

... o 

o 

o 

o 

o 

o 

d. participated in practicum courses, internships, co-ops, clinical 

assignments, or field experiences?. 

... o 

o 

o 

o 

o 

o 

e. interacted with people from different backgrounds or cultures?. 

... o 

o 

o 

o 

o 

o 

f. performed community service/volunteer activities?. 

... o 

o 

o 

o 

o 

o 

g. participated in research or creative projects with a faculty member? . 

... o 

o 

o 

o 

o 

o 

h. presented a project or paper with a faculty member? . 

... o 

o 

o 

o 

o 

o 

i. attended a play, concert, art exhibit, lecture, or other cultural event? . 

.. o 

o 

o 

o 

o 

o 

j. participated in a meeting or activities of a professional organization? . 

... o 

o 

o 

o 

o 

o 

k. participated in study abroad or other international activities? . 

... o 

o 

o 

o 

o 

o 

1 . participated in leadership training or had a leadership role? . 

.. o 

o 

o 

o 

o 

o 

m attended a WKU athletic event? . 

... o 

o 

o 

o 

o 

o 

n. worked with a faculty member on a project outside class 9 

...o 

o 

o 

o 

o 

o 

o. been contacted by your advisor concerning your degree program?. 

...o 

o 

o 

o 

o 

o 

p. made a class presentation?. 

... o 

o 

o 

o 

o 

o 

q. worked with classmates outside of class to prepare class assignments?. 

....o 

o 

o 

o 

o 

o 

r. had serious conversations with someone of a different race or ethnicity 
than your own?. 

... o 

o 

o 

o 

o 

o 

s. had serious conversations with students who are very different from you 
terms of their religious beliefs, political opinions, or personal values?. 

in 

... O 

o 

o 

o 

o 

o 

t. discussed ideas from your readings or classes with faculty members 
outside class?. 

.. O 

o 

o 

o 

o 

o 


8a. While at WKU, on average, HOW MANY HOURS PER WEEK have you spent working in a job? 

0 0 1-5 O 6-10 O 11-15 O 16-20 O 21-25 O 26-30 O 30+ 

b. HOW MANY HOURS PER WEEK do you spend doing volunteer work? (unpaid activities - religious organizations, 
hospitals, schools, off-campus charitable activities) 

O 0 0 1-2 O 3-4 O 5-9 O 10 or more 

9a. How many of the courses in your major included knowledge None 

or activities that will help you become a contributing citizen?. O 

b. How many of your non-major courses included knowledge 

or activities that will help you become a contributing citizen?.O 

c. How many of the courses in your major required community 

involvement, volunteerism or other out-of-class learning activities?.O 

d. How many of your non-major courses required community 

involvement, volunteerism or other out-of-class learning activities?.O 

10. During the past year, how often have you participated in a community-based project as part of your course work? 
O Very Often O Often O Sometimes O Never 


To what extent has your course work at WKU: 

Very Little 

Some 

Quite a Bit 

Very Much 

a. contributed to your voting in local, state, or national elections?. 

.o 

O 

o 

o 

b. affected your contribution to the welfare of your community?. 

. o 

O 

o 

o 

c. helped you synthesize and organize ideas, information, or experiences?. 

. o 

o 

o 

o 

d. helped you analyze the basic elements of an idea, experience, or theory?. 

. o 

o 

o 

o 


o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 


:. What is your ethnicity? 13. 

What is your 

14. What is your 

15. At which campus have you taken the majority 

O 

Non-Resident Alien 

gender? 

classification? 

(51% or more) of your work while at WKU? 

O 

Amer. Ind/Alaskan Native 

O Female 

O Freshman 

O Main Campus 

O Glasgow 

O 

Asian/Pacific Islander 

O Male 

O Sophomore 

O South Campus 

O Owensboro 

O 

Black 


O Junior 

Elizabethtown 

C Radcliff 

O 

Hispanic 


O Senior 

Fort Knox 

O On-line 

O 

White 


Grad. Student 
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